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In 2001 Astumian l] published a very simple game which can be described by a Markov chain 
with absorbing initial and final states. In August 2004 Piotrowski and Sladowski |3| asserted that 
Astumian's analysis was flawed. However, as was shown by Astumian Q, this statement was wrong. 

' In this comment the properties of Markov chains corresponding to games that are more general than 

' that studied by Astumian, are investigated. 
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I. INTRODUCTION 
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[ The present note was initiated by the revisited Astumian's paradox. In August 2004 Piotrowski and Sladowski 
' asserted that Astumian's analysis was flawed. However, as shown by Astumian this statement was wrong. Since 
' the analysis of the problem in a slightly more general frame than it was done earlier could be a good exercise for 
(~| ■ graduate students, we came to the conclusion that it might be useful to publish our elementary considerations about 
' the properties of Markov chains corresponding to Astumian type games. 

. For entirely didactic reasons, in Sections II and III we present a brief summary of definitions and statements which 
are needed for the analysis of the Astumian type Markov chains. In Section IV we analyze the properties of such 
chains and determine the probabilities of losing and winning. Conclusions are made in Section V. 
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.$H ! II. PRELIMINARIES 

f~| Let Jy = {1,2,..., N} be a finite set of positive integers, and Z = {0, 1, . . .} be a set of non-negative integers. 

Qh' Denote by n € Z the random variable which assumes the elements of A/". We say that the sequence {^„} forms a 
Markov chain if for all n G Z and for all possible values of random variables the equation 

^{Cn = iICo = «0,6 = «1, • ■ • = = ^{61 = j|Cn-l = in-l} (1) 

I is fulfilled. If ^„ = j then the process is said to be in state Sj at the nth (discrete time instant) step. The states 

^-H ' 5i,iS2, . . . ,Sn define the space of states of the process. The probability distribution V{^o — i}, i G Af of the random 

^sj variable is called the initial distribution and the conditional probabilities Vl^n = j\S,n-i = i} are called transition 

^-H probabilities. If £,n-i = i and ^„ = j, then we say that the process made a transition Si Sj at the nth step. The 

, Markov chain is homogeneous if the transition probabilities are independent of rt G In this case we may write 

C/3 ' Vi^n = j\£.n-l = = = W^j, 

o ; 

■ ^ . and it obviously holds that 

>>: 

ViGA/-. (2) 



In what follows we shall consider only homogeneous Markov chains. We would like to emphasize that the transition 
probability matrix 



f wii wi2 ■■■ Win\ 

W21 W22 ■ ■ ■ W2N 
\WN1 WN2 ■■■ WnnJ 



(3) 



which is a stochastic matrix, and the initial distribution pi = V{^q = i}, i G Af determine the random process 
uniquely. For the sake of simplicity, we assume that the process is a random walk of an abstract object, called particle 
on the space of states Si,S2, ■ ■ ■ , Sn . The nth step transition probability 



'P{£.rn+7i = j\£.rn = «} = Wij{n) 



(4) 
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satisfies the following equation: 



N 



(5) 



where 



k=l 



r + s ~ n. 



It is to note that Wij (n) is the probability that at the nth step the particle is in the state Sj provided that at n = 
it was in the state Si. From Eq. Q we obtain that 



AT 



N 



i{n) = ^Wife Wkj{n - 1) = '^Wik{n - 1) 



Wkj, 



k=l fc=l 

and by using the rules of matrix multiplication we arrive at 

w(n) = w w(n — 1) w(n — 1) w = w", 

where 



(6) 



w(n) 



( wii{n) Wi2(n) ••• WiN[n)\^ 
W2i{n) W22{n) ■■■ W2N{n) 



\wNi{n) WN2{n) ■■■ WNNin)J 



(7) 



and 



w(0) 



/I ••• 0\ 
1 ■•• 



\o ... Ij 



is the N X N unit matrix. 

Making use of the total probability theorem we can determine the absolute probabilities Pj{n) as follows: 



N 



(n) = ^ Pi w,j (n), j e JV, 



(8) 



where pi = V{S,o — i} is the initial probability. Clearly, Pj{n) is the probability that the particle is in the state Sj at 
the nth step. Introducing the row vector 



Eq. © can be rewritten in the form: 



P= {pi,P2,---,Pn} 



p{n) = pw{n) = w^'^^ (n) jf-^"^ , 



(9) 



(10) 



where the upper index T indicates the transpose of matrix w(n) and vector p defined by iQ and ©, respectively. If 
the process starts from the state 5^, then 



and 



P= {0i,02,...,lj,...,0Ar}, 

Pi[n) = {wii{n),Wi2{n), ■ ■ .,WiN{n)}. 
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III. TYPES OF STATES AND ASYMPTOTIC BEHAVIOR 



A. Basic definitions 



In order to use clear notions, we introduce several well-known definitions. If there is an integer n > such that 
Wjk{n) > 0, then we say the state Sk can be reached from the state Sj. If Sk can be reached from Sj and Sj can 
be reached from Sk, then Sj and Sk are connected states. Obviously, if Sj and Sk are not connected, then either 
Wjk{n) = 0, or Wkj{n) = 0. The set of states which are connected forms a class of equivalence. A Markov chain is 
called irreducible if every state can be reached from every state i.e., the entire state space consists of only one class 
of equivalence. In other words, the Markov chain is irreducible when all of the states are connected. 

The probability (n) of passage from Si to Sj in exactly n steps, that is, without passing through Sj before the 
nth step, is given by 

fij{n)= ^ ••• Wj„_ij. (11) 

There exists an important relationship between the probabilities Wij{n) and fij{n) which is easy to prove. The 
relationship is given by 

n 

Wij{n) = '^^ fij{k) Wjj{n — k), 'in&Z. (12) 
fe=i 

One has to note that the expressions Wjj{Q) = 1 are the diagonal elements of the unit matrix w(0). 

The proof of (|12|1 is immediate upon applying the total probability rule. The particle passes from Si to Sj in n 
steps if, and only if, it passes from Si to Sj for the first time in exactly k steps, k = 1,2, ... ,n, and then passes 
from Sj to Sj in the remaining n ~ k steps. These "paths" are disjoint events, and their probabilities are given by 
fij{k) Wjj(n — k). Summing over k one obtains the equation H12|) . 

Let us introduce the generating functions 

OO CX) 

Vijiz) = ^ ("-) 2" and i^i][z) = ^ Wij{n) z". (13) 

n— 1 n=l 

Taking into account that Wjj (0) = 1 , from Eq. 112(1 we obtain 

Uij [z) = Lpij {z) [I + UJjj (z)] , (14) 



and from this 



so we have 



and in particular 



^..W = T^^, (15) 
1 + ^jj{z) 



,^1 ^ + 2ln=lWjj{n) 



„=1 l-ErT=l/jjW 



Fij defined by H16|l is the probability that a particle starting its walk from Si passes through the state Sj at least 
once. Clearly, Fa = Fi is the probability of returning to Si at least once. 

More generally, the probability Fijik) that a particle starting its walk from Si passes through Sj at least k times 
is given by 



F,,(k) 



E 

n=l 



F,,{k - 1) = F,, F,,{k - 1) 
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In particular, the probability of returning to Si at least k times is given by Fu{k) = {Fa)''. Its limit 

0, if Fa < 1, 

1, ifF,;, =l 



Rii = lim {FiiY 

k—^oo 



is the probability of returning to Si infinitely often. It follows from the previous relationship that the probability that 
a particle starting its walk from <Sj passes through Sj infinitely many times is 



R 



so that 




Ri 



We say that Si is a return state or a nonreturn state according as Fj > or Fj = 0. As a further definition, we say 
that Si is a recurrent state or a nonrecurrent state according as Fj = 1 or < < 1. A nonrecurrent state is often 
called a transient state. 

The state Si is called periodic with period ^ if a return to Si can occur only at steps £, 2£, 'M, . . . and ^ > 1 is the 
greatest integer with this property. If n is not divisible by £, then Wij (n) = 0. If the period of each state is equal to 
1, i.e., a i = 1, then the Markov chain is called aperiodic. In the sequel we are dealing with aperiodic Markov chains. 

A set C of states in a Markov chain is closed if it is impossible to move out from any state of C to any state outside 



C by one-step transitions, i.e. 



,(1) 



a Si G C and Sj ^ C. In this case Wij (n) = obviously holds for every 



n G Z. If a single state Si forms a closed set, then we call this an absorbing state, and we have wa = 1. 

The states of a closed set C are recurrent states since the return probability Fi for any state Si € C is equal to 1. 
Therefore, the set of recurrent states is denoted by C. ^ The set of states having return probabilities F^ < 1 is the 

if Sj is an absorbing state, then 



set of transient states and it is denoted by T. Obviously, if Si £ T and Sj G C, 
Fij is the probability that a particle starting at Si is finally absorbed at Sj . 

Let fij be the passage time of a particle from the state Si to the state Sj, taking values m = 1,2 
probabilities fij{m). If 



with 



fiM) = Pi,3 = 1> 



m— 1 



then the expected passage time Tij = E{fy } from Si to Sj is defined by 

'dip^jiz)' 



m=l 



dz 



while if Fij < 1, one says that Vij = oo with probability 1 — Fy , i.e., if Fij < 1, then the expected passage time 



00. If the state Sj = Si and it is recurrent, i.e., if Fa = Fj = 1, then the expectation 



^Wii} = "1 fiii'm) 



m=l 



d(pii{z) 
dz 



(18) 



is called mean recurrent time. If /ij = oo, then we say that Si is a recurrent null-state, whereas if fii < oo, then we 
say that <S, is a recurrent non-null-state. If Fi < 1, i.e., the state Si is transient, then 1 — Fj is the probability that 
the recurrence time is infinitely long, and so fii = oo. 

We say that the recurrent state Si is ergodic, if it is not a null-state and is aperiodic, that is, if Fj = 1, m < oo and 

e = i. 



^ The set C can be decomposed into mutually disjoint closed sets Ci,C2, ■ ■ ■ ,Cr such that from any state of a given set all states of that 
set and no others can be reached . States Ci , C2 , • • • , Cr can be reached from T, but not conversely. 
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B. Asymptotic behavior 

The first statement is very simple, hence it is given without proof. If Sj is a transient or a recurrent null-state, 
then for any arbitrary Si 

lim WiJn) = (19) 

n — >oo 

holds. 

If Si and Sj are recurrent aperiodic states due to the same closed set, then 

lim Wij{n) = — , (20) 



irrespective of Si. ^ 

If i = j, then we have from Eq. (|14() the formula 



1 - fjjiz) 

Substituting this into (|14|l we obtain the following expression: 



^..(^) = T^^- (21) 



^,,(z) = ^,,(z) fl + -£Mi±\ 



1 - ipjj{z)' 



By using Tauber's Theorem we can state that 



lim (1 - z) ^ ""i^, - lim wy(n). (23) 



zfl l — (pjj{z) n^oo 

Since Si and Sj are aperiodic recurrent states due to the same closed set 

lim(^y(z) = lim if jj{z) = 1, 

i.e., the limit value we have to determine 

1 - z 



lim ■ . 

znl- (Pjj(z) 



Applying L'Hospital's rule we find that 

1 - z 



(22) 



lim , 

zn 1 - ipjj(z) Lp'{i) 

and thus we obtain H20() . This completes the proof. 

As a generalization we would like to consider the case when Si is a transient state {Si G T) and Sj is an aperiodic 
recurrent state due to the closed set C. It can be shown that 

lim «;„(n) = (24) 

n^oc fj,j 

where Fij is the probability that a particle starting from Si will ultimately reach and stay in the state Sj £ C. In 
other words, Fij is the absorption probability that satisfies the following system of equations: 

= + w^kFkj. V S, e T. (25) 
Sk^r 



^ In order to prove the limit relationship 1201 Tauber's Theorem is used instead of the lemma by Erdos-Feller-Kac. 
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Clearly, if T U C contains all of the possible states of the particle, then 

E = 1- (26) 

The proof of (|24(l follows immediately from H22(l . Since 

we obtain the limit relationship H24|l . 

Finally, we would like to present a brief classification of Markov chains. 

• A Markov chain is called irreducible if and only if all its states form a closed set and there is no other closed set 
contained in it. 

• A Markov chain is called ergodic if the probability distributions 

N 
k=\ 

always converge to a limiting distribution p-,- which is independent of the initial distribution Pj(0), that is, when 
lim„^oo Pi ('T') — Vj^ S ^ ■ AH states of a finite, aperiodic irreducible Markov chain are ergodic. 

• The probability distribution "pi is a stationary distribution of a Markov chain if, when we choose it as an 

initial distribution all the distributions pi (n) will coincide with p'f*'^ . Every stationary distribution of a Markov 
chain satisfies the following system of linear equations: 

Pj"''*^ " E^i^*' ^^^^ ^Pf*^ = ^' 

i 3 

(st) 

and conversely, each solution pj of this system is a stationary distribution of the Markov chain, if it is a 
probability distribution. 

It is to mention that some parts of this short summary is based on the small but excellent book by Takacs 



IV. MARKOV CHAINS WITH ABSORBING STATES 



In this section we are going to deal with Markov chains containing two absorbing states Si and Sn, and N — 2 
transient states. In this case, the Markov chain is reducible and aperiodic. The set of its states is the union of two 
closed sets Ci = {^i} and C2 — {Sn}, and of the set of transient states T = {52,53, . . . ,5Ar_i} The states Si and 
Sn can be reached from each state of T but the converse doesn't hold, no state of T can be reached from the states 
Si and Sn- The states of T are non-recurrent since the particle leaves the set never to return to it. In contrary, the 
states of Ci and C2 are ergodic. 



A. Chains of five states 



Let us assume that the transition matrix w has the following form: 
















W21 


W22 


W23 











W31 


W33 













W43 


W44 


W45 


V 











1 / 



(27) 



where 



1. 



7 



The particle, which starts his walk from one of the states Sf, {i = 2,3,4), is captured when it enters the states iSi 
or 55. By using the foregoing formulae for Fn and _Fi5, we can immediately obtain the capture probabilities by the 
absorbing states Si and 1S5, respectively. In order to have a direct insight into the nature of the process, we derive 
the backward equations for the probabilities Wij{n). Clearly, 



Wij {n 
W2j {n 
W3j{n 
Wij {n 
W5j{n 



= Sij, (28) 

= W21 wij{n - 1) + 1^22 W2j{n - 1) + W2Z wzj{n - 1), (29) 

= W32 W2j{n - 1) + W33 W3,j{n - 1) + W34 Wij{n - 1), (30) 

= u'43 W3j{n - 1) + W44 Wij{n - 1) + w^j{n - 1), (31) 

= -^Si, (32) 



and by introducing tlic generating function 

00 

gij{z) = 6ij + ^ Wij{n) = 5ij +u)ij{z), \z\<l, (33) 



Tl=l 

we obtain the following system of equations: 

1 



gij{z) = Si^ - 



z 

92j{z) = 62j + ZW21 gij{z) + ZW22 92j{z) + ZW23 g3j{z), 
93j{z) = Ssj + ZW32 g2j{z) + ZW33 g3j{z) + ZW34 gij{z), 

gij{z) = 64j+W43g3j{z) + ZW44g4j{z) + Z'W45g5j{z), 
95j{z) = S5J 

I — z 

This can be simplified and rewritten in the form: 

{I - ZW22) g2j(z) - ZW23 g3j{z) = 52j + W21- (34) 

L — Z 

-ZW32 g2j{z) + {1 - ZW33) g3j{z) - ZW34: g4j{z) = (53j, (35) 

z 

-ZW43 93j{z) + (1 - W44) 9ij{z) = 64j + W4bY~^ ^^^^ 

After elementary algebra, we can determine all the generating functions gij{z), — 1,2,3,4,5), nevertheless we 

are now interested only in those functions which correspond to processes starting from the state ^Ss. In this case we 
have 

_ Z^ W32W2l{l - W44Z) , . 

'''^'^ - l^z W) ' ^ ^ 

_ W32^(l - W44{Z)Z) 

^^^^'^ - — W) — ' ^ ^ 

n tr\ - - W22Z){1 - W44Z) 

W43{1-W22Z) 

^''^'^ = — W) — ' ^ ^ 

Z^ W34W4^{\ - W22Z) , , 

'''^'^ = l^z W) ' ^ ^ 

where 

D{z) = (1 - W44Z) [(1 - W22Z){\. - W33Z) - W23W32Z^\ - (1 - W22z)w34W43Z^ . (42) 

Applying Tauber's Theorem we obtain that 

r I \ V ^^ \ I \ ^321*^21(1-^44) 

Jim u;3i(n) = hm(l - z) 53i(^) = , (43) 
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lim W3j{n) = lim(l - z) g^^iz) = 0, j = 2, 3, 4, (44) 

n — ^oo z]! 

and 

lim W35[n) = lim(l - z) .935(2) = — — . (45) 

Performing the substitutions 

W22 = 1 - W21 - W23, W33 = 1 - W32 - W34 W44 = 1 - W43 - W45, 

we have 

/ N W32Ui2l(w43 + u;45) 

'^3i[oo) = — -, (46) 

W34W45{W21 +W23)+ «^32W2l(W43 + W45) 

and 

. ^ 1^34^45(^21 + 'u;23) ..„^ 
w^35(oo) = — -. (47) 

^34^45(^21 + W23) + W32W2l(W43 + U'45) 

It is elementary to show that 

F31 = <(f3i(l) = W3i(oo) and F35 = (^35(1) = ^35(00). (48) 

In order to prove these equations, let us take into account relationship (|15|l and write 

/ X ^31(2) 531 (^^) 

"^31(2^) = — r-- = r^, 

l + cjii(z) gii[z) 

and 

/ X W35(z) 535 (z) 

l + CJ55(z) 555(2:) 

Since 



511(2) = 555(2) 

we have 



1 



¥'31(2:) = (1 - 2;)53i(2:) and (^35(2) = (1 - 2)535(2). (49) 



Comparing (gSJ and gs)) with we see that Eqs. (0SI) are true. 

It is convenient to write the absorption probabilities i^3i and i^35 in the form: 



^31--—, and F35 = -— , (50) 
1 + r 1 + r 



where 



^ _ W34W45(W21 + W23) 
W32W2l(w43 + W45) 

and we see immediately that i^3i + ^35 = 1, as expected. 

It seems to be worthwhile to study the history of a particle starting its random walk from the state ^3. Let us 
consider a trap containing a special ladder with 5 rungs. Each rung corresponds to a given state of the Markov chain 
under investigation. The process starts when a particle enters (say,) on the third rung of the ladder, i.e., in the state 
53. Once the particle has entered, it is free to move up and down the rungs randomly . Fig. 1 illustrates this random 
walk. If the particle reaches the states either iSi or 1S5, it is absorbed. (If the random walk is considered as a game. 
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55 
54 
53 
52 

5i 



FIG. 1: Illustration ol the random walk on a ladder ol five rungs 




20 40 60 80 100 
number of steps 

FIG. 2: Dependencies of the probabilities 1031(71), 1033 (n) and W35(n) on the number of steps n 



then the absorption state with probabihty smaller than 1/2 is the 
matrix 



'winning" state.) Having chosen the transition 



/I \ 

4/36 24/36 8/36 ' 

5/36 29/36 2/36 

4/36 24/36 8/36 

V 1 / 



(52) 



we calculated the dependencies of probabilities w^i{n),wz'i(n) and ^35(71) on the number of steps n. The results of 
calculation are shown in Fig. 2. We see that the probability to find the particle after n ~ 100 steps in the transient 
state 1S3 is practically zero. The same holds for the transient states 1S2 and 1S4. After n « 100 steps the particle is 
absorbed either in Si with probability W3i(100) « F31 — 5/9 or in 1S5 with probability ^35(100) k, ^"35 = 4/9. 

It is instructive to determine also the probabilities -^32,^33 and ^34. As a reminder, we note that F^j is the 
probability that a particle starting from 1S3 passes through Sj, {j — 2,3,4) at least once. By using the transition 
matrix 1)52(1 we obtain the following values: i^32 = 15/19, ^33 = 11/12 and i^34 = 6/11. Fig. 3 shows the histogram of 
these probabilities. It is evident that passing through either iSi or 55 at least once means that the particle is absorbed. 
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probability 

0.8 
0.6 
0.4 
0.2 



I 



3^1 3^2 3^3 3^4 3^5 

FIG. 3: Probabilities that a particle passes through the state Sj, {j = 1, 2, 3, 4, 5) at least once provided that it started from <S3 

As expected in the present case, the probabiUty F32, that the particle starting from 1S3 returns to 53 at least once, is 
nearly 1. It is to mention that the two absorbing states Si and ^5 are recurrent since Fu = F55 = 1. 

In what follows we would like to deal with the determination of the absorption time probability. Denote by the 
number of steps leading to the absorption of a particle starting its random walk from the state Si. By definition, 
fii{n) and /i5(n) are the probabilities that the particle starting from the state Si, [i = 2, 3, 4) is absorbed exactly at 
the nth step in iSi or in iSs, respectively. Hence we can write that 



It is easy to prove that 



From (|12|l one obtains 



V{n = n}^ T,(n) = fa{n) + Uin), i = 2,3,4. 



Ti{n) = Wii{n) - Wii{n - 1) + Wi^in) - Wi^in - 1), V n > 1. 



(53) 
(54) 



Wii{n) = /ii(fc) wn{n - fc), 

n 

Wi5{n) = ^fi5{k)w55{n-k), 



k=l 



and by taking into account that 



one has 



w. 



= ^ fii{k) and Wi^in) = ^ fisik). 



k=l 



fc=i 



It follows immediately from these equations that 

fii{n) ^ Wii{n) " Wii{n - 1) and fisin) ^ Wi^in) - Wi^in - 1), 

and this completes the proof. The absorption time probabilities Ti{n), {i = 2,3,4) can be determined by the 
"forward" equations: 



fii{n) = '^Wii{n - 1) 



e=2 
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and 



fi5{n) ^^wu{n - 1) 



£=2 



By using these expressions one can write 



Ti{n) ^^Wii{n - l)[wn + w^] , 



i=2 



which in the case of w defined by H27|) has the foUowing form: 

Ti{n) = Wi2{n ~ 1) W2\ + Wi^(n - 1) W45. 
For the sake of completeness, we would like to show that 



In the case of Eq. H53|l we see that 



i5 1 



and by using the expression if^ we find In the case of Eq. (I55|l 



Wii{n - 1) 



(55) 



(56) 



(57) 



[5u + LOiiil)] [ wn + Wf5] = '^guil) [ wn + w^] = Fn + = 1. 



£=2 



«=2 




<T2> = 15.7 
<T3>=19.0 
<T4>=9.3 



^ 0.2 
n 
o 

°-0.15 

I 0.1 

■° 0.05 
o 

^ 

10 20 30 40 50 
number of steps 

FIG. 4: Absorption time probabilities of a particle starting its random walk from the state Si, (i = 2, 3, 4) 

Using the transition matrix w given by H52|l . we calculated the dependence of the probability Ti{n) on the number 
of steps n. The results are seen in Fig. 4. As expected, if the starting state is S^, then the probability T^{n) varies 




12 



differently with the step number as the probabiHties T2{n) and T4(n). It is characteristic the probabiUties have a 
rather long tail. Since Ti(n) is the probability that a particle starting from Si is absorbed exactly in the nth step, 
the expectation and the standard deviation of the absorption time ti are given by 



E{r.} = 



n T.,{n) =< Ti >, 



(58) 



and 



1/2 



(59) 



For a transition matrix of the form H52|) these values are presented in the Table I. 



TABLE I: Expectations and the standard deviations of the absorption time 





S2 


.S3 


54 


E{r,} 


15.7 


19.0 


9.3 




16.3 


16.4 


13.3 



B. Properties of the absorption probability F31 



As it has been shown, F^i is the probability that a particle starting its random walk from the state S3 is finally 
absorbed in the state Si. If F31 > 1/2, then Si is called a "losing" state, while if F31 < 1/2, then it is a "winning" 
state. The game is "fair" when F31 = 1/2, i.e. when the equation 

W32W21 (w43 + W45) = W34W45 (w2i + "^23) (60) 

is fulfilled as it follows from Eq. H51|) . 

Astumian proposed two transition matrices, namely 



Wi 



/ 1 \ 

4/36 24/36 8/36 

5/36 29/36 2/36 

4/36 24/36 8/36 

\ 1 / 



W2 



/ 1 \ 

4/36 24/36 8/36 

5/36 29/36 2/36 

4/36 24/36 8/36 
V 1 / 



resulting in the absorption probability F31 — 5/9 > 1/2 and showed that the arithmetic mean of these two matrices 

w = - (Wl + Wl) = 



/ 1 \ 

9/72 53/72 10/72 

9/72 53/72 10/72 

9/72 53/36 10/72 

V 1 / 



^ There is no need to deal separately with the absorption probability F35 since F31 + _F35 = 1. 
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brings about the probability F^i = 9/19 < 1/2, i.e., in this case the state Si becomes "winning" state. This property 
of the transition matrix H27|l is general if the diagonal entries of the matrix are different from zero. By using a simple 
example we would like to demonstrate this statement. 
Let us choose the transition matrix in the following form: 



/I 

a 1 — a — 6 
b 

\0 





b 

l~a-b- 
a + X 




One obtains immediately that 



where 



F31 = H{x) 



\ 



X a + X 

1 — a — 5 b — X 
1 / 



ah 



(61) 



ab + {a + x)(b — x) 



(62) 



0<a<l, 0<5<1, 0<a + fe<l and - a < x < mm{b,l - a - b). 
If X = ov x = b — a, then the game is "fair", i.e., F31 — 1/2. The function H{x) assumes its minimal value at 

1 



and this value is 



-^(^min) 



4afe 
Jab+Ja+bj^ 



Introducing the notation x — Xmm + y one has 

H{xmin + y) = Jiy) 

Choosing y according to the inequalities 

Xl = Xmm + y > 

i.e., y > \xmin\ and a ^b one finds that 

ff(xi) = H{X2) > I 



1/2, if a = 6, 

< 1/2, if a 7^ 6. 

Aab+ (a + 6)2 -Ay'^' 
and X2 = Xrmn - y < b - a, 



and H 



X2 



< 



1 



Evidently, there are infinitely many pairs of transition matrices which result in probabilities of losing in the state Si 
but the arithmetic means of corresponding pairs bring about probabilities of winning in the state Si. 

For the sake of illustration in Fig. 5 the probability F31 = H{x) vs. x curve is plotted by the values a = 1/4 and 
& = 1/8. The black points Ci, C2 and c correspond to the probabilities 



ir(xi = 1/16) = H"(2;2 = -3/16) = 8/13 and H 



2(2^1 +2^2) 



-1/16 



8/17, 



respectively. It seems to be not superfluous to write down the corresponding transition matrices: 



Wl 



/ 1 \ 

4/16 10/16 2/16 

2/16 9/16 5/16 

5/16 10/16 1/16 

V 1 / 



W2 



/ 1 \ 

4/16 10/16 2/16 

2/16 13/16 1/16 

1/16 10/16 5/16 
\ 1 / 



(63) 
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and 



gO.75 

^ 0.7 

0.65 

I' 0.6 

I 0.55 
o 



0.5 



a=1/4, b=1/8 
H(x^in)=8/1 7*0.47 

V X^in=-1/16 

\ 




— •— — 





-0.2 -0.15 -0.1 -0.05 0.05 0.1 

X values 

FIG. 5: Dependence of .Fsi = Hix) on the parameter x of the transition matrix 1611 when a = 1/4 and b = 1/8 



/ 1 \ 

4/16 10/16 2/16 

2/16 11/16 3/16 

3/16 10/16 3/16 
V 1 / 



(64) 



By choosing y values in the allowed interval, we can construct infinitely many transition matrices with just described 
properties. 




0.6 
gO.5 

1 — 

m 

^ 0.4 

I 0.3 

I 0.2 
o 

Q-0.1 


10 20 30 40 50 
number of steps 

FIG. 6: Dependence of 1031(71) on the number of steps n in the case of p = 1/2 



Let us now define a Markov chain with transition matrix w randomly chosen from wi and W2 defined by (|63|) . In 
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0.2 0.4 0.6 0.8 1 
probability of choosing the gamewi 

FIG. 7: Dependence of the absorption probabilities F3i(ra) on the chance p moving the particle at a given step according to the transition 
matrix wi 

this case 

'^(n) = [pwi + (1 — p)w2] X w(n — 1), 

i.e., 

w(n) = [pwi + (1 - p)w2]" . (65) 

In Fig. 6 the dependencies of the absorption probabilities W3i(n) ^ on the number of steps n are shown when the 
transition matrices are wi, W2 and w, respectively. The last one corresponds to the random selection of the entries 
from wi and W2 with probability p — 1/2. Obviously, not all values of p G [0, 1] bring about a "winning" game, i.e., 
an absorption probability less than 1/2. 

Taking into account the transition matrices Wi and W2 defined by (|^ . we determined the dependence of F^i on p. 
As seen in Fig. 7, there is a well defined subinterval [pi,P2] G [0, 1] containing the p values which result in absorption 
probabilities F31 smaller than 1/2. In the present case we obtained that pi = 0.25 and p2 — 0.75. 

V. CONCLUSIONS 

It has been shown that the random walk of a particle defined by the stochastic transition matrix of a Markov chain 
is equivalent to an Astumian type game if the diagonal entries of the matrix are different from zero and the first 
(wii) as well as the last (wj^pf) entries are equal to 1. By using a simple example, we have proved that there are 
infinitely many pairs of transition matrices which result in absorption probabilities in the state Si larger than 1/2 
but the arithmetic means of the corresponding pairs lead to probabilities smaller than 1/2. 
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As seen before, W3i(n) is the first entry of the third row of the matrix w(n). 



